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-- The MAILING DATE of this communication appears on the cover sheet with the correspondence address ~ 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MO NTH (S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1.136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1)^ Responsive to communication(s) filed on 20 November 2003 . 
2a)D This action is FINAL. 2b)^ This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) ^ Claim(s) 1-3. 7-22,26 and 28-30 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) ^ Claim(s) 1-3. 7-22.26 and 28-30 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10) S The drawing(s) filed on 07 November 2003 is/are: a)^ accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

11) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 
Priority under 35 U.S.C. §§119 and 120 

12) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 
a)DAII b)D Some*c)D None of: 

Certified copies of the priority documents have been received. 
Certified copies of the priority documents have been received in Application No. . 



1. D 

2. D 
3D 



Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

1 3) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 1 1 9(e) (to a provisional application) 

since a specific reference was included in the first sentence of the specification or in an Application Data Sheet. 
37 CFR 1 .78. 

a) □ The translation of the foreign language provisional application has been received. 

14) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121 since a specific 

reference was included in the first sentence of the specification or in an Application Data Sheet. 37 CFR 1 .78. 



Attachment(s) 

1) □ Notice of References Cited (PTO-892) 4) □ Interview Summary (PTO-413) Paper No(s). 

2) 0 Notice of Draftsperson's Patent Drawing Review (PTO-948) 5) Notice of Informal Patent Application (PTO-1 52) 

3) □ Information Disclosure Statement(s) (PTO-1 449) Paper No(s) . 6) □ Other: 
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DETAILED ACTION 

This action is in response to communication filed 1 1/20/03. 

Drawings 

1. The drawings were received on 1 1/7/03. These drawings are accepted. 

Specification 

2. The objection to the specification for lacking a "Summary of the Invention" is 
maintained, because the application fails to be in accordance with the accepted guidelines of 
current U.S. practice. Applicant is directed to 37 CFR 1.73, which states, "A brief summary of 
the invention indicating its nature and substance, which may include a statement of the object of 
the invention, should precede the detailed description. . It is noted that the brief summary 
should be inserted prior to the detailed description. Applicant is directed to MPEP § 608.01(d). 

Claim Rejections - 35 USC §103 

3. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

4. Claims 1-3, 7-22, and 26 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
U.S. Patent No. 6,227,968 to Suzuki et al. (hereinafter Suzuki) and U.S. Patent No. 6,256,033 to 
Nguyen (hereinafter Nguyen). 
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Referring to claims 1 and 26, Suzuki teaches a method and computer program product 
that receives audio data having a beat and forming beat data based on the audio data (col. 7, lines 
24-34). Suzuki teaches determining a gesture window within which a gesture (dance move) 
should occur, based on a specified time window relative to the beat data, and Suzuki plays the 
audio data and obtains gesture data during a time that the audio data is being played. See col. 7, 
lines 35-50. Also, see col. 1, lines 9-13, which describes how the player is to perform the action 
in time with the rhythm. 

Suzuki segments the gesture data according to the specified time window (time between 
beats), and Suzuki automatically determines information related to a gesture occurring within the 
specified timing window (i.e. if the appropriate dance move/gesture occurred within the timing 
window for score keeping). See col. 7, lines 25-55 and col. 10, line 30 - col. 11, line 37. 

Suzuki does not teach that the gesture data is obtained through video data, wherein the 
video data is segmented to create a video clip of time including specified time window. 

However, Nguyen discloses a method and a computer program product for recognizing 
gestures contained in video data that segments video data to create a video clip based on timing 
data that indicates a specified timing window within which a gesture will occur. See col. 2, lines 
23-25, which states, "The frame captures the person in the action of performing the gesture at 
one moment in time" and col. 2, lines 28-34, which states, "These sequence of frame data sets 
taken over a period of time. . ." The method determines information related to a gesture 
occurring in the video clip only at the specified timing. See col. 3, lines 11-14, which describes 
"determining particular coordinates of the sub ject at a particular time ", (emphasis added). 
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It would have been obvious to one of ordinary skill in the art to substitute the gesture 
recognition method (step-on base of a floor panel) in the dance game of Suzuki with the video 
gesture recognition method of Nguyen, such that the video data is segmented into video clips of 
specified timing according to beat data as supported by the beat timing window of Suzuki, 
because the step-on base of Suzuki only captures the movements of the player's feet, and does 
not recognize the movements of the rest of the body as intended in Suzuki (col. 1, lines 44-5 1), 
wherein the video gesture recognition method of Nguyen captures the movements of the entire 
body. 

Referring to claims 14, Suzuki teaches a system with an audio part that receives audio 
data having a beat and forming beat data based on the audio data (col. 7, lines 24-34). Suzuki 
teaches determining a gesture window within which a gesture (dance move) should occur, based 
on a specified time window relative to the beat data, and Suzuki plays the audio data and obtains 
gesture data during a time that the audio data is being played. See col. 7, lines 35-50. Also, see 
col. 1, lines 9-13, which describes how the player is to perform the action in time with the 
rhythm. 

Suzuki segments the gesture data according to the specified time window (time between 
beats), and Suzuki automatically determines information related to a gesture occurring within the 
specified timing window (i.e. if the appropriate dance move/gesture occurred within the timing 
window for score keeping). See col. 7, lines 25-55 and col. 10, line 30 - col. 1 1, line 37. 

Suzuki does not teach that the gesture data is obtained through video data, wherein the 
video data is segmented to create a video clip of time including specified time window. 
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However, Nguyen discloses a system with a processor (Fig. 1, 102) for recognizing 
gestures contained in video data that has a temporal segmentor that segments video data to create 
a video clip based on timing data that indicates a specified timing window within which a gesture 
will occur. See col, 2, lines 23-25, which states, "The frame captures the person in the action of 
performing the gesture at one moment in time" and col. 2, lines 28-34, which states, "These 
sequence of frame data sets taken over a period of time. . ." The method determines information 
related to a gesture occurring in the video clip only at the specified timing. See col. 3, lines 11- 
14, which describes "determining particular coordinates of the subject at a particular time ", 
(emphasis added). Nguyen also discloses a recognition engine, in communication with the 
temporal segmentor, to determine if the video clip contains a predefined gesture, only within a 
specified timing window. See col. 1, lines 17-23 and col. 3, lines 1 1-14. 

It would have been obvious to one of ordinary skill in the art to substitute the gesture 
recognition system (step-on base of a floor panel) in the dance game of Suzuki with the video 
gesture recognition system of Nguyen, such that the video data is segmented into video clips of 
specified timing according to beat data as supported by the beat timing window of Suzuki, 
because the step-on base of Suzuki only captures the movements of the player's feet, and does 
not recognize the movements of the rest of the body as intended in Suzuki (col. 1, lines 44-5 1), 
wherein the video gesture recognition system of Nguyen captures the movements of the entire 
body. 
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Referring to claims 2-3, the combination of Suzuki and Nguyen, supra, describes how 
Hidden Markov Models are used to determine a probability that each of a plurality of predefined 
gestures are performed within a timing window. See col. 5, lines 13-45 of Nguyen. 

Referring to claim 7, 17-18, and 21, the dance game of Suzuki teaches displaying a target 
gesture to be performed by the subject. The target gesture is a dance move. See col. 12, lines 
57-61. The target gesture is displayed on the display subsystem (monitor; col. 12, line 59). 

Referring to claims 8-10, the combination of Suzuki and Nguyen, supra, teaches each 
video clip contains video frames (Nguyen at col. 7, line 22), and in each frame, the moving 
regions are identified (Nguyen at col. 8, lines 58-61). Feature vectors (array of key points) are 
generated for each video frame of the video clip. See Nguyen at col. 8, line 49 - col. 9, line 46, 
which describes how significant positional coordinates are extracted from each frame to make a 
comparison to the known gesture coordinates. 

Referring to claims 1 1-12 and 19-20, the method of Suzuki and Nguyen, supra, generates 
and displays a score based on whether a target movement (gesture) was performed. See Suzuki 
at col. 1 1, lines 37-41. The object of the Suzuki dance game is to achieve the highest score by 
performing the appropriate gestures. 

Referring to claims 13 and 22, the combination of Suzuki and Nguyen, supra, discloses 
that the recognition engine is configured to recognize predefined gestures and that determining if 
the video clip contains a target gesture includes generating a gesture probability vector (array) 
having a plurality of elements, each element being associated with one of a predefined gestures 
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and representing a probability that the video clip contains each of the associated predefined 
gestures. See col. 11, lines 30-49 and col. 10, lines 18-37. 

Referring to claim 15, the recognition engine of Suzuki and Nguyen, supra, includes a 
plurality of Hidden Markov Models. See Nguyen at col. 5, lines 13-18. 

Referring to claim 16, the system of Suzuki and Nguyen, supra, includes a video source 
(Nguyen at Fig. 2, 200), in communication with the temporal segmentor, to provide the video 
data to the temporal segmentor. 

5. Claims 28-30 are rejected under 35 U.S.C. 103(a) as being unpatentable over Nguyen and 
Suzuki. 

Referring to claim 28, Nguyen teaches an A/V processing system that includes a video 
source (camera; Fig. 2, 200). Nguyen discloses a computer program product for recognizing 
gestures contained in video data, comprising instruction operable to cause a programmable 
processor, in communication with the video source to segment the video clip and automatically 
determine if the video clip contains a predefined gesture. See col. 1, lines 17-23. 

Nguyen determines a gesture window within which a gesture should occur. See col. 2, 
lines 23-25, which states, "The frame captures the person in the action of performing the gesture 
at one moment in time" and col. 2, lines 28-34, which states, "These sequence of frame data sets 
taken over a period of time. . ." The method determines information related to a gesture 
occurring in the video clip only at the specified timing. See col. 3, lines 11-14, which describes 
"determining particular coordinates of the subject at a particular time ", (emphasis added). 
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Nguyen does not explicitly teach an audio source to provide audio data having a beat 
from which beat data is extracted, such that the a specified time window of the segments are 
based on the beat data or obtaining the video data during a time that the audio signal is being 
produced. However, Suzuki teaches a dance game, which is intended to make the player use his 
entire body to create rhythm sensations (col. 1, lines 47-50). The dance game of Suzuki teaches 
that audio data is received and the beat data is extracted to create a specified timing window (col. 
7, lines 24-34). Suzuki determines if a predefined gesture (dance move) is performed by 
detecting dance steps on a step-on base only within a specified timing window related to the beat 
data. For example, see col. 14, lines 6-18. It would have been obvious to one of ordinary skill in 
the art at the time of the invention to modify the gesture recognition method of Nguyen to be 
used in the dance game of Suzuki, such that the video data is obtained during a time that the 
audio data is played and the specified timing window for seeing if a gesture occurred is related to 
beat data extracted from audio data as supported by Suzuki. One would have been so motivated 
in order to compare all movements of a player (i.e. arms) to an appropriate dance move because 
the step-on base of Suzuki only captures the movements of the player's feet, and does not 
recognize the movements of the rest of the body as intended in Suzuki. 

Referring to claim 29, Nguyen teaches the computer program product includes instruction 
operable to cause the programmable processor to perform a Hidden Markov Model process to 
determine if the video clip contains the predefined gesture. See col. 5, lines 13-18. 

Referring to claim 30, Nguyen discloses a display (monitor; Fig. 2, 208) to display 
information based on whether the video clip contains the predefined gesture. See col. 6, lines 
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36-44, which describes how figures on the display can be augmented if the gesture is recognized 
as a predefined gesture. 

Response to Arguments 

6. Applicant's arguments filed 1 1/20/03 have been fully considered but they are not 
persuasive. 

7. In response to applicant's argument that the examiner's conclusion of obviousness is 
based upon improper hindsight reasoning, it must be recognized that any judgment on 
obviousness is in a sense necessarily a reconstruction based upon hindsight reasoning. But so 
long as it takes into account only knowledge which was within the level of ordinary skill at the 
time the claimed invention was made, and does not include knowledge gleaned only from the 
applicant's disclosure, such a reconstruction is proper. See In re McLaughlin, 443 F.2d 1392, 
170 USPQ 209 (CCPA 1971). 

Applicant argues that nothing in the dance game of Suzuki teaches video acquisition and 
that Nguyen teaches nothing about a dance game. However, Suzuki teaches that the dance game 
uses a form of gesture recognition, and Nguyen teaches a more sophisticated form of gesture 
recognition available at the time of the present invention. One of ordinary skill in the art would 
have readily recognized the advantage of the video gesture recognition of Nguyen over the step- 
on base gesture recognition of Suzuki at the time of the invention. Therefore, one of ordinary 
skill in the art would have been motivated to substitute the gesture recognition (step-on base) of 
Suzuki with the more sophisticated gesture recognition of Nguyen for a more accurate capture of 
dance moves (i.e. capturing the entire body instead of just the feet). 
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Applicant argues that Suzuki does not teach analyzing information automatically within 
the video clip. However, Nguyen teaches analyzing the video clip, and Suzuki teaches looking 
for a gesture within a specified timing window (i.e. according to a rhythm). Therefore, the 
combination of Suzuki and Nguyen, described above, teach each limitation claimed. 

Applicant argues with respect to claims 10-11 and 22, that Nguyen does not teach a 
feature vector or a gesture probability vector. See Nguyen at col. 8, line 49 - col. 9, line 46, 
which describes how significant positional coordinates are extracted from each frame to make a 
comparison to the known gesture coordinates, wherein the gesture coordinates are stored in an 
array. Therefore, the array of key points makes up the feature vector. Col. 1 1, lines 30-49 and 
col. 10, lines 18-37, describes a probability array (vector). 



Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Shawn M. Becker whose telephone number is 703-305-7756. 
The examiner can normally be reached on M-Th 8:00 - 5:30 and alternating Fridays. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, John W. Cabeca can be reached on 703-305-3 116. The fax phone number for the 
organization where this application or proceeding is assigned is 703-746-7239. 

Any inquiry of a general nature or relating to the status of this application or proceeding 
should be directed to the receptionist whose telephone number is 703-30^3900. 



Conclusion 
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